{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b518b04cbfe0"
      },
      "source": [
        "##### Copyright 2020 The TensorFlow Authors."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "cellView": "form",
        "id": "906e07f6e562"
      },
      "outputs": [],
      "source": [
        "#@title Licensed under the Apache License, Version 2.0 (the \"License\");\n",
        "# you may not use this file except in compliance with the License.\n",
        "# You may obtain a copy of the License at\n",
        "#\n",
        "# https://www.apache.org/licenses/LICENSE-2.0\n",
        "#\n",
        "# Unless required by applicable law or agreed to in writing, software\n",
        "# distributed under the License is distributed on an \"AS IS\" BASIS,\n",
        "# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.\n",
        "# See the License for the specific language governing permissions and\n",
        "# limitations under the License."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "fb291b62b1aa"
      },
      "source": [
        "# Training and evaluation with the built-in methods"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b1820d9bdfb9"
      },
      "source": [
        "<table class=\"tfo-notebook-buttons\" align=\"left\">\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://www.tensorflow.org/guide/keras/train_and_evaluate\"><img src=\"https://www.tensorflow.org/images/tf_logo_32px.png\" />View on TensorFlow.org</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://colab.research.google.com/github/tensorflow/docs/blob/snapshot-keras/site/en/guide/keras/train_and_evaluate.ipynb\"><img src=\"https://www.tensorflow.org/images/colab_logo_32px.png\" />Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a target=\"_blank\" href=\"https://github.com/keras-team/keras-io/blob/master/guides/training_with_built_in_methods.py\"><img src=\"https://www.tensorflow.org/images/GitHub-Mark-32px.png\" />View source on GitHub</a>\n",
        "  </td>\n",
        "  <td>\n",
        "    <a href=\"https://storage.googleapis.com/tensorflow_docs/docs/site/en/guide/keras/train_and_evaluate.ipynb\"><img src=\"https://www.tensorflow.org/images/download_logo_32px.png\" />Download notebook</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8d4ac441b1fc"
      },
      "source": [
        "## Setup"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0472bf67b2bf"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "from tensorflow import keras\n",
        "from tensorflow.keras import layers"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e2c78442fa34"
      },
      "source": [
        "## Introduction\n",
        "\n",
        "This guide covers training, evaluation, and prediction (inference) models\n",
        "when using built-in APIs for training & validation (such as `model.fit()`,\n",
        "`model.evaluate()`, `model.predict()`).\n",
        "\n",
        "If you are interested in leveraging `fit()` while specifying your\n",
        "own training step function, see the guide\n",
        "[\"customizing what happens in `fit()`\"](https://www.tensorflow.org/guide/keras/customizing_what_happens_in_fit/).\n",
        "\n",
        "If you are interested in writing your own training & evaluation loops from\n",
        "scratch, see the guide\n",
        "[\"writing a training loop from scratch\"](https://www.tensorflow.org/guide/keras/writing_a_training_loop_from_scratch/).\n",
        "\n",
        "In general, whether you are using built-in loops or writing your own, model training &\n",
        "evaluation works strictly in the same way across every kind of Keras model --\n",
        "Sequential models, models built with the Functional API, and models written from\n",
        "scratch via model subclassing.\n",
        "\n",
        "This guide doesn't cover distributed training. For distributed training, see\n",
        "our [guide to multi-gpu & distributed training](/guides/distributed_training/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4e270faa413e"
      },
      "source": [
        "## API overview: a first end-to-end example\n",
        "\n",
        "When passing data to the built-in training loops of a model, you should either use\n",
        "**NumPy arrays** (if your data is small and fits in memory) or **`tf.data Dataset`\n",
        "objects**. In the next few paragraphs, we'll use the MNIST dataset as NumPy arrays, in\n",
        "order to demonstrate how to use optimizers, losses, and metrics.\n",
        "\n",
        "Let's consider the following model (here, we build in with the Functional API, but it\n",
        "could be a Sequential model or a subclassed model as well):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "170a6a18b2a3"
      },
      "outputs": [],
      "source": [
        "inputs = keras.Input(shape=(784,), name=\"digits\")\n",
        "x = layers.Dense(64, activation=\"relu\", name=\"dense_1\")(inputs)\n",
        "x = layers.Dense(64, activation=\"relu\", name=\"dense_2\")(x)\n",
        "outputs = layers.Dense(10, activation=\"softmax\", name=\"predictions\")(x)\n",
        "\n",
        "model = keras.Model(inputs=inputs, outputs=outputs)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e6d5724a90ab"
      },
      "source": [
        "Here's what the typical end-to-end workflow looks like, consisting of:\n",
        "\n",
        "- Training\n",
        "- Validation on a holdout set generated from the original training data\n",
        "- Evaluation on the test data\n",
        "\n",
        "We'll use MNIST data for this example."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "8b55b3903edb"
      },
      "outputs": [],
      "source": [
        "(x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()\n",
        "\n",
        "# Preprocess the data (these are NumPy arrays)\n",
        "x_train = x_train.reshape(60000, 784).astype(\"float32\") / 255\n",
        "x_test = x_test.reshape(10000, 784).astype(\"float32\") / 255\n",
        "\n",
        "y_train = y_train.astype(\"float32\")\n",
        "y_test = y_test.astype(\"float32\")\n",
        "\n",
        "# Reserve 10,000 samples for validation\n",
        "x_val = x_train[-10000:]\n",
        "y_val = y_train[-10000:]\n",
        "x_train = x_train[:-10000]\n",
        "y_train = y_train[:-10000]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "77a84eb1985b"
      },
      "source": [
        "We specify the training configuration (optimizer, loss, metrics):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "26a7f1819796"
      },
      "outputs": [],
      "source": [
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(),  # Optimizer\n",
        "    # Loss function to minimize\n",
        "    loss=keras.losses.SparseCategoricalCrossentropy(),\n",
        "    # List of metrics to monitor\n",
        "    metrics=[keras.metrics.SparseCategoricalAccuracy()],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ef28150b1eaa"
      },
      "source": [
        "We call `fit()`, which will train the model by slicing the data into \"batches\" of size\n",
        "\"batch_size\", and repeatedly iterating over the entire dataset for a given number of\n",
        "\"epochs\"."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0b92f67b105e"
      },
      "outputs": [],
      "source": [
        "print(\"Fit model on training data\")\n",
        "history = model.fit(\n",
        "    x_train,\n",
        "    y_train,\n",
        "    batch_size=64,\n",
        "    epochs=2,\n",
        "    # We pass some validation for\n",
        "    # monitoring validation loss and metrics\n",
        "    # at the end of each epoch\n",
        "    validation_data=(x_val, y_val),\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0a1b698c6e39"
      },
      "source": [
        "The returned \"history\" object holds a record of the loss values and metric values\n",
        "during training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "a20b8f5b9fcc"
      },
      "outputs": [],
      "source": [
        "history.history"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6105b646df66"
      },
      "source": [
        "We evaluate the model on the test data via `evaluate()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "69f524a93f9d"
      },
      "outputs": [],
      "source": [
        "# Evaluate the model on the test data using `evaluate`\n",
        "print(\"Evaluate on test data\")\n",
        "results = model.evaluate(x_test, y_test, batch_size=128)\n",
        "print(\"test loss, test acc:\", results)\n",
        "\n",
        "# Generate predictions (probabilities -- the output of the last layer)\n",
        "# on new data using `predict`\n",
        "print(\"Generate predictions for 3 samples\")\n",
        "predictions = model.predict(x_test[:3])\n",
        "print(\"predictions shape:\", predictions.shape)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f19d074eb88c"
      },
      "source": [
        "Now, let's review each piece of this workflow in detail."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f3669f026d14"
      },
      "source": [
        "## The `compile()` method: specifying a loss, metrics, and an optimizer\n",
        "\n",
        "To train a model with `fit()`, you need to specify a loss function, an optimizer, and\n",
        "optionally, some metrics to monitor.\n",
        "\n",
        "You pass these to the model as arguments to the `compile()` method:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "eb7a8deb494c"
      },
      "outputs": [],
      "source": [
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),\n",
        "    loss=keras.losses.SparseCategoricalCrossentropy(),\n",
        "    metrics=[keras.metrics.SparseCategoricalAccuracy()],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c4061c977ac3"
      },
      "source": [
        "The `metrics` argument should be a list -- your model can have any number of metrics.\n",
        "\n",
        "If your model has multiple outputs, you can specify different losses and metrics for\n",
        "each output, and you can modulate the contribution of each output to the total loss of\n",
        "the model. You will find more details about this in the section **\"Passing data to\n",
        "multi-input, multi-output models\"**.\n",
        "\n",
        "Note that if you're satisfied with the default settings, in many cases the optimizer,\n",
        "loss, and metrics can be specified via string identifiers as a shortcut:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6444839ff300"
      },
      "outputs": [],
      "source": [
        "model.compile(\n",
        "    optimizer=\"rmsprop\",\n",
        "    loss=\"sparse_categorical_crossentropy\",\n",
        "    metrics=[\"sparse_categorical_accuracy\"],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5493ab963254"
      },
      "source": [
        "For later reuse, let's put our model definition and compile step in functions; we will\n",
        "call them several times across different examples in this guide."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "31c3e3c70f06"
      },
      "outputs": [],
      "source": [
        "def get_uncompiled_model():\n",
        "    inputs = keras.Input(shape=(784,), name=\"digits\")\n",
        "    x = layers.Dense(64, activation=\"relu\", name=\"dense_1\")(inputs)\n",
        "    x = layers.Dense(64, activation=\"relu\", name=\"dense_2\")(x)\n",
        "    outputs = layers.Dense(10, activation=\"softmax\", name=\"predictions\")(x)\n",
        "    model = keras.Model(inputs=inputs, outputs=outputs)\n",
        "    return model\n",
        "\n",
        "\n",
        "def get_compiled_model():\n",
        "    model = get_uncompiled_model()\n",
        "    model.compile(\n",
        "        optimizer=\"rmsprop\",\n",
        "        loss=\"sparse_categorical_crossentropy\",\n",
        "        metrics=[\"sparse_categorical_accuracy\"],\n",
        "    )\n",
        "    return model\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "21b19c0a6a85"
      },
      "source": [
        "### Many built-in optimizers, losses, and metrics are available\n",
        "\n",
        "In general, you won't have to create from scratch your own losses, metrics, or\n",
        "optimizers, because what you need is likely already part of the Keras API:\n",
        "\n",
        "Optimizers:\n",
        "\n",
        "- `SGD()` (with or without momentum)\n",
        "- `RMSprop()`\n",
        "- `Adam()`\n",
        "- etc.\n",
        "\n",
        "Losses:\n",
        "\n",
        "- `MeanSquaredError()`\n",
        "- `KLDivergence()`\n",
        "- `CosineSimilarity()`\n",
        "- etc.\n",
        "\n",
        "Metrics:\n",
        "\n",
        "- `AUC()`\n",
        "- `Precision()`\n",
        "- `Recall()`\n",
        "- etc."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "d7abc0339980"
      },
      "source": [
        "### Custom losses\n",
        "\n",
        "There are two ways to provide custom losses with Keras. The first example creates a\n",
        "function that accepts inputs `y_true` and `y_pred`. The following example shows a loss\n",
        "function that computes the mean squared error between the real data and the\n",
        "predictions:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "cc4edd47bb5a"
      },
      "outputs": [],
      "source": [
        "def custom_mean_squared_error(y_true, y_pred):\n",
        "    return tf.math.reduce_mean(tf.square(y_true - y_pred))\n",
        "\n",
        "\n",
        "model = get_uncompiled_model()\n",
        "model.compile(optimizer=keras.optimizers.Adam(), loss=custom_mean_squared_error)\n",
        "\n",
        "# We need to one-hot encode the labels to use MSE\n",
        "y_train_one_hot = tf.one_hot(y_train, depth=10)\n",
        "model.fit(x_train, y_train_one_hot, batch_size=64, epochs=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "25b9fa7941ca"
      },
      "source": [
        "If you need a loss function that takes in parameters beside `y_true` and `y_pred`, you\n",
        "can subclass the `tf.keras.losses.Loss` class and implement the following two methods:\n",
        "\n",
        "- `__init__(self)`: accept parameters to pass during the call of your loss function\n",
        "- `call(self, y_true, y_pred)`: use the targets (y_true) and the model predictions\n",
        "(y_pred) to compute the model's loss\n",
        "\n",
        "Let's say you want to use mean squared error, but with an added term that\n",
        "will de-incentivize  prediction values far from 0.5 (we assume that the categorical\n",
        "targets are one-hot encoded and take values between 0 and 1). This\n",
        "creates an incentive for the model not to be too confident, which may help\n",
        "reduce overfitting (we won't know if it works until we try!).\n",
        "\n",
        "Here's how you would do it:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b09463a8c568"
      },
      "outputs": [],
      "source": [
        "class CustomMSE(keras.losses.Loss):\n",
        "    def __init__(self, regularization_factor=0.1, name=\"custom_mse\"):\n",
        "        super().__init__(name=name)\n",
        "        self.regularization_factor = regularization_factor\n",
        "\n",
        "    def call(self, y_true, y_pred):\n",
        "        mse = tf.math.reduce_mean(tf.square(y_true - y_pred))\n",
        "        reg = tf.math.reduce_mean(tf.square(0.5 - y_pred))\n",
        "        return mse + reg * self.regularization_factor\n",
        "\n",
        "\n",
        "model = get_uncompiled_model()\n",
        "model.compile(optimizer=keras.optimizers.Adam(), loss=CustomMSE())\n",
        "\n",
        "y_train_one_hot = tf.one_hot(y_train, depth=10)\n",
        "model.fit(x_train, y_train_one_hot, batch_size=64, epochs=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b2141cc075a6"
      },
      "source": [
        "### Custom metrics\n",
        "\n",
        "If you need a metric that isn't part of the API, you can easily create custom metrics\n",
        "by subclassing the `tf.keras.metrics.Metric` class. You will need to implement 4\n",
        "methods:\n",
        "\n",
        "- `__init__(self)`, in which you will create state variables for your metric.\n",
        "- `update_state(self, y_true, y_pred, sample_weight=None)`, which uses the targets\n",
        "y_true and the model predictions y_pred to update the state variables.\n",
        "- `result(self)`, which uses the state variables to compute the final results.\n",
        "- `reset_states(self)`, which reinitializes the state of the metric.\n",
        "\n",
        "State update and results computation are kept separate (in `update_state()` and\n",
        "`result()`, respectively) because in some cases, results computation might be very\n",
        "expensive, and would only be done periodically.\n",
        "\n",
        "Here's a simple example showing how to implement a `CategoricalTruePositives` metric,\n",
        "that counts how many samples were correctly classified as belonging to a given class:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "05d6a6e7022d"
      },
      "outputs": [],
      "source": [
        "class CategoricalTruePositives(keras.metrics.Metric):\n",
        "    def __init__(self, name=\"categorical_true_positives\", **kwargs):\n",
        "        super(CategoricalTruePositives, self).__init__(name=name, **kwargs)\n",
        "        self.true_positives = self.add_weight(name=\"ctp\", initializer=\"zeros\")\n",
        "\n",
        "    def update_state(self, y_true, y_pred, sample_weight=None):\n",
        "        y_pred = tf.reshape(tf.argmax(y_pred, axis=1), shape=(-1, 1))\n",
        "        values = tf.cast(y_true, \"int32\") == tf.cast(y_pred, \"int32\")\n",
        "        values = tf.cast(values, \"float32\")\n",
        "        if sample_weight is not None:\n",
        "            sample_weight = tf.cast(sample_weight, \"float32\")\n",
        "            values = tf.multiply(values, sample_weight)\n",
        "        self.true_positives.assign_add(tf.reduce_sum(values))\n",
        "\n",
        "    def result(self):\n",
        "        return self.true_positives\n",
        "\n",
        "    def reset_states(self):\n",
        "        # The state of the metric will be reset at the start of each epoch.\n",
        "        self.true_positives.assign(0.0)\n",
        "\n",
        "\n",
        "model = get_uncompiled_model()\n",
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),\n",
        "    loss=keras.losses.SparseCategoricalCrossentropy(),\n",
        "    metrics=[CategoricalTruePositives()],\n",
        ")\n",
        "model.fit(x_train, y_train, batch_size=64, epochs=3)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4bca8e959cda"
      },
      "source": [
        "### Handling losses and metrics that don't fit the standard signature\n",
        "\n",
        "The overwhelming majority of losses and metrics can be computed from `y_true` and\n",
        "`y_pred`, where `y_pred` is an output of your model. But not all of them. For\n",
        "instance, a regularization loss may only require the activation of a layer (there are\n",
        "no targets in this case), and this activation may not be a model output.\n",
        "\n",
        "In such cases, you can call `self.add_loss(loss_value)` from inside the call method of\n",
        "a custom layer. Losses added in this way get added to the \"main\" loss during training\n",
        "(the one passed to `compile()`). Here's a simple example that adds activity\n",
        "regularization (note that activity regularization is built-in in all Keras layers --\n",
        "this layer is just for the sake of providing a concrete example):"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b494d47437a0"
      },
      "outputs": [],
      "source": [
        "class ActivityRegularizationLayer(layers.Layer):\n",
        "    def call(self, inputs):\n",
        "        self.add_loss(tf.reduce_sum(inputs) * 0.1)\n",
        "        return inputs  # Pass-through layer.\n",
        "\n",
        "\n",
        "inputs = keras.Input(shape=(784,), name=\"digits\")\n",
        "x = layers.Dense(64, activation=\"relu\", name=\"dense_1\")(inputs)\n",
        "\n",
        "# Insert activity regularization as a layer\n",
        "x = ActivityRegularizationLayer()(x)\n",
        "\n",
        "x = layers.Dense(64, activation=\"relu\", name=\"dense_2\")(x)\n",
        "outputs = layers.Dense(10, name=\"predictions\")(x)\n",
        "\n",
        "model = keras.Model(inputs=inputs, outputs=outputs)\n",
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),\n",
        "    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        ")\n",
        "\n",
        "# The displayed loss will be much higher than before\n",
        "# due to the regularization component.\n",
        "model.fit(x_train, y_train, batch_size=64, epochs=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "aaebb5829011"
      },
      "source": [
        "You can do the same for logging metric values, using `add_metric()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "aa58091be092"
      },
      "outputs": [],
      "source": [
        "class MetricLoggingLayer(layers.Layer):\n",
        "    def call(self, inputs):\n",
        "        # The `aggregation` argument defines\n",
        "        # how to aggregate the per-batch values\n",
        "        # over each epoch:\n",
        "        # in this case we simply average them.\n",
        "        self.add_metric(\n",
        "            keras.backend.std(inputs), name=\"std_of_activation\", aggregation=\"mean\"\n",
        "        )\n",
        "        return inputs  # Pass-through layer.\n",
        "\n",
        "\n",
        "inputs = keras.Input(shape=(784,), name=\"digits\")\n",
        "x = layers.Dense(64, activation=\"relu\", name=\"dense_1\")(inputs)\n",
        "\n",
        "# Insert std logging as a layer.\n",
        "x = MetricLoggingLayer()(x)\n",
        "\n",
        "x = layers.Dense(64, activation=\"relu\", name=\"dense_2\")(x)\n",
        "outputs = layers.Dense(10, name=\"predictions\")(x)\n",
        "\n",
        "model = keras.Model(inputs=inputs, outputs=outputs)\n",
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(learning_rate=1e-3),\n",
        "    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        ")\n",
        "model.fit(x_train, y_train, batch_size=64, epochs=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f3c18154d057"
      },
      "source": [
        "In the [Functional API](https://www.tensorflow.org/guide/keras/functional/),\n",
        "you can also call `model.add_loss(loss_tensor)`,\n",
        "or `model.add_metric(metric_tensor, name, aggregation)`.\n",
        "\n",
        "Here's a simple example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0e19afe78b3a"
      },
      "outputs": [],
      "source": [
        "inputs = keras.Input(shape=(784,), name=\"digits\")\n",
        "x1 = layers.Dense(64, activation=\"relu\", name=\"dense_1\")(inputs)\n",
        "x2 = layers.Dense(64, activation=\"relu\", name=\"dense_2\")(x1)\n",
        "outputs = layers.Dense(10, name=\"predictions\")(x2)\n",
        "model = keras.Model(inputs=inputs, outputs=outputs)\n",
        "\n",
        "model.add_loss(tf.reduce_sum(x1) * 0.1)\n",
        "\n",
        "model.add_metric(keras.backend.std(x1), name=\"std_of_activation\", aggregation=\"mean\")\n",
        "\n",
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
        "    loss=keras.losses.SparseCategoricalCrossentropy(from_logits=True),\n",
        ")\n",
        "model.fit(x_train, y_train, batch_size=64, epochs=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b06d48035369"
      },
      "source": [
        "Note that when you pass losses via `add_loss()`, it becomes possible to call\n",
        "`compile()` without a loss function, since the model already has a loss to minimize.\n",
        "\n",
        "Consider the following `LogisticEndpoint` layer: it takes as inputs\n",
        "targets & logits, and it tracks a crossentropy loss via `add_loss()`. It also\n",
        "tracks classification accuracy via `add_metric()`."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "d56d2c504258"
      },
      "outputs": [],
      "source": [
        "class LogisticEndpoint(keras.layers.Layer):\n",
        "    def __init__(self, name=None):\n",
        "        super(LogisticEndpoint, self).__init__(name=name)\n",
        "        self.loss_fn = keras.losses.BinaryCrossentropy(from_logits=True)\n",
        "        self.accuracy_fn = keras.metrics.BinaryAccuracy()\n",
        "\n",
        "    def call(self, targets, logits, sample_weights=None):\n",
        "        # Compute the training-time loss value and add it\n",
        "        # to the layer using `self.add_loss()`.\n",
        "        loss = self.loss_fn(targets, logits, sample_weights)\n",
        "        self.add_loss(loss)\n",
        "\n",
        "        # Log accuracy as a metric and add it\n",
        "        # to the layer using `self.add_metric()`.\n",
        "        acc = self.accuracy_fn(targets, logits, sample_weights)\n",
        "        self.add_metric(acc, name=\"accuracy\")\n",
        "\n",
        "        # Return the inference-time prediction tensor (for `.predict()`).\n",
        "        return tf.nn.softmax(logits)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0698f3c98cbe"
      },
      "source": [
        "You can use it in a model with two inputs (input data & targets), compiled without a\n",
        "`loss` argument, like this:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0f6842f2bbe6"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "inputs = keras.Input(shape=(3,), name=\"inputs\")\n",
        "targets = keras.Input(shape=(10,), name=\"targets\")\n",
        "logits = keras.layers.Dense(10)(inputs)\n",
        "predictions = LogisticEndpoint(name=\"predictions\")(logits, targets)\n",
        "\n",
        "model = keras.Model(inputs=[inputs, targets], outputs=predictions)\n",
        "model.compile(optimizer=\"adam\")  # No loss argument!\n",
        "\n",
        "data = {\n",
        "    \"inputs\": np.random.random((3, 3)),\n",
        "    \"targets\": np.random.random((3, 10)),\n",
        "}\n",
        "model.fit(data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "328b021aa6b8"
      },
      "source": [
        "For more information about training multi-input models, see the section **Passing data\n",
        "to multi-input, multi-output models**."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0536882b969c"
      },
      "source": [
        "### Automatically setting apart a validation holdout set\n",
        "\n",
        "In the first end-to-end example you saw, we used the `validation_data` argument to pass\n",
        "a tuple of NumPy arrays `(x_val, y_val)` to the model for evaluating a validation loss\n",
        "and validation metrics at the end of each epoch.\n",
        "\n",
        "Here's another option: the argument `validation_split` allows you to automatically\n",
        "reserve part of your training data for validation. The argument value represents the\n",
        "fraction of the data to be reserved for validation, so it should be set to a number\n",
        "higher than 0 and lower than 1. For instance, `validation_split=0.2` means \"use 20% of\n",
        "the data for validation\", and `validation_split=0.6` means \"use 60% of the data for\n",
        "validation\".\n",
        "\n",
        "The way the validation is computed is by taking the last x% samples of the arrays\n",
        "received by the fit call, before any shuffling.\n",
        "\n",
        "Note that you can only use `validation_split` when training with NumPy data."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "232fd59c751b"
      },
      "outputs": [],
      "source": [
        "model = get_compiled_model()\n",
        "model.fit(x_train, y_train, batch_size=64, validation_split=0.2, epochs=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "42969af7ce01"
      },
      "source": [
        "## Training & evaluation from tf.data Datasets\n",
        "\n",
        "In the past few paragraphs, you've seen how to handle losses, metrics, and optimizers,\n",
        "and you've seen how to use the `validation_data` and `validation_split` arguments in\n",
        "fit, when your data is passed as NumPy arrays.\n",
        "\n",
        "Let's now take a look at the case where your data comes in the form of a\n",
        "`tf.data.Dataset` object.\n",
        "\n",
        "The `tf.data` API is a set of utilities in TensorFlow 2.0 for loading and preprocessing\n",
        "data in a way that's fast and scalable.\n",
        "\n",
        "For a complete guide about creating `Datasets`, see the\n",
        "[tf.data documentation](https://www.tensorflow.org/guide/data).\n",
        "\n",
        "You can pass a `Dataset` instance directly to the methods `fit()`, `evaluate()`, and\n",
        "`predict()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "3bf4ded224f8"
      },
      "outputs": [],
      "source": [
        "model = get_compiled_model()\n",
        "\n",
        "# First, let's create a training Dataset instance.\n",
        "# For the sake of our example, we'll use the same MNIST data as before.\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
        "# Shuffle and slice the dataset.\n",
        "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
        "\n",
        "# Now we get a test dataset.\n",
        "test_dataset = tf.data.Dataset.from_tensor_slices((x_test, y_test))\n",
        "test_dataset = test_dataset.batch(64)\n",
        "\n",
        "# Since the dataset already takes care of batching,\n",
        "# we don't pass a `batch_size` argument.\n",
        "model.fit(train_dataset, epochs=3)\n",
        "\n",
        "# You can also evaluate or predict on a dataset.\n",
        "print(\"Evaluate\")\n",
        "result = model.evaluate(test_dataset)\n",
        "dict(zip(model.metrics_names, result))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "421d16914ce3"
      },
      "source": [
        "Note that the Dataset is reset at the end of each epoch, so it can be reused of the\n",
        "next epoch.\n",
        "\n",
        "If you want to run training only on a specific number of batches from this Dataset, you\n",
        "can pass the `steps_per_epoch` argument, which specifies how many training steps the\n",
        "model should run using this Dataset before moving on to the next epoch.\n",
        "\n",
        "If you do this, the dataset is not reset at the end of each epoch, instead we just keep\n",
        "drawing the next batches. The dataset will eventually run out of data (unless it is an\n",
        "infinitely-looping dataset)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "273c5dff16b4"
      },
      "outputs": [],
      "source": [
        "model = get_compiled_model()\n",
        "\n",
        "# Prepare the training dataset\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
        "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
        "\n",
        "# Only use the 100 batches per epoch (that's 64 * 100 samples)\n",
        "model.fit(train_dataset, epochs=3, steps_per_epoch=100)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f2dcd180da7b"
      },
      "source": [
        "### Using a validation dataset\n",
        "\n",
        "You can pass a `Dataset` instance as the `validation_data` argument in `fit()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "bf4f3d78e69a"
      },
      "outputs": [],
      "source": [
        "model = get_compiled_model()\n",
        "\n",
        "# Prepare the training dataset\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
        "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
        "\n",
        "# Prepare the validation dataset\n",
        "val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))\n",
        "val_dataset = val_dataset.batch(64)\n",
        "\n",
        "model.fit(train_dataset, epochs=1, validation_data=val_dataset)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2e7f0ebf5f1d"
      },
      "source": [
        "At the end of each epoch, the model will iterate over the validation dataset and\n",
        "compute the validation loss and validation metrics.\n",
        "\n",
        "If you want to run validation only on a specific number of batches from this dataset,\n",
        "you can pass the `validation_steps` argument, which specifies how many validation\n",
        "steps the model should run with the validation dataset before interrupting validation\n",
        "and moving on to the next epoch:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f47342fed069"
      },
      "outputs": [],
      "source": [
        "model = get_compiled_model()\n",
        "\n",
        "# Prepare the training dataset\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train))\n",
        "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
        "\n",
        "# Prepare the validation dataset\n",
        "val_dataset = tf.data.Dataset.from_tensor_slices((x_val, y_val))\n",
        "val_dataset = val_dataset.batch(64)\n",
        "\n",
        "model.fit(\n",
        "    train_dataset,\n",
        "    epochs=1,\n",
        "    # Only run validation using the first 10 batches of the dataset\n",
        "    # using the `validation_steps` argument\n",
        "    validation_data=val_dataset,\n",
        "    validation_steps=10,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "67b4418e9f26"
      },
      "source": [
        "Note that the validation dataset will be reset after each use (so that you will always\n",
        "be evaluating on the same samples from epoch to epoch).\n",
        "\n",
        "The argument `validation_split` (generating a holdout set from the training data) is\n",
        "not supported when training from `Dataset` objects, since this feature requires the\n",
        "ability to index the samples of the datasets, which is not possible in general with\n",
        "the `Dataset` API."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8160beb766a0"
      },
      "source": [
        "## Other input formats supported\n",
        "\n",
        "Besides NumPy arrays, eager tensors, and TensorFlow `Datasets`, it's possible to train\n",
        "a Keras model using Pandas dataframes, or from Python generators that yield batches of\n",
        "data & labels.\n",
        "\n",
        "In particular, the `keras.utils.Sequence` class offers a simple interface to build\n",
        "Python data generators that are multiprocessing-aware and can be shuffled.\n",
        "\n",
        "In general, we recommend that you use:\n",
        "\n",
        "- NumPy input data if your data is small and fits in memory\n",
        "- `Dataset` objects if you have large datasets and you need to do distributed training\n",
        "- `Sequence` objects if you have large datasets and you need to do a lot of custom\n",
        "Python-side processing that cannot be done in TensorFlow (e.g. if you rely on external libraries\n",
        "for data loading or preprocessing).\n",
        "\n",
        "\n",
        "## Using a `keras.utils.Sequence` object as input\n",
        "\n",
        "`keras.utils.Sequence` is a utility that you can subclass to obtain a Python generator with\n",
        "two important properties:\n",
        "\n",
        "- It works well with multiprocessing.\n",
        "- It can be shuffled (e.g. when passing `shuffle=True` in `fit()`).\n",
        "\n",
        "A `Sequence` must implement two methods:\n",
        "\n",
        "- `__getitem__`\n",
        "- `__len__`\n",
        "\n",
        "The method `__getitem__` should return a complete batch.\n",
        "If you want to modify your dataset between epochs, you may implement `on_epoch_end`.\n",
        "\n",
        "Here's a quick example:\n",
        "\n",
        "```python\n",
        "from skimage.io import imread\n",
        "from skimage.transform import resize\n",
        "import numpy as np\n",
        "\n",
        "# Here, `filenames` is list of path to the images\n",
        "# and `labels` are the associated labels.\n",
        "\n",
        "class CIFAR10Sequence(Sequence):\n",
        "    def __init__(self, filenames, labels, batch_size):\n",
        "        self.filenames, self.labels = filenames, labels\n",
        "        self.batch_size = batch_size\n",
        "\n",
        "    def __len__(self):\n",
        "        return int(np.ceil(len(self.filenames) / float(self.batch_size)))\n",
        "\n",
        "    def __getitem__(self, idx):\n",
        "        batch_x = self.filenames[idx * self.batch_size:(idx + 1) * self.batch_size]\n",
        "        batch_y = self.labels[idx * self.batch_size:(idx + 1) * self.batch_size]\n",
        "        return np.array([\n",
        "            resize(imread(filename), (200, 200))\n",
        "               for filename in batch_x]), np.array(batch_y)\n",
        "\n",
        "sequence = CIFAR10Sequence(filenames, labels, batch_size)\n",
        "model.fit(sequence, epochs=10)\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2a28343b1967"
      },
      "source": [
        "## Using sample weighting and class weighting\n",
        "\n",
        "With the default settings the weight of a sample is decided by its frequency\n",
        "in the dataset. There are two methods to weight the data, independent of\n",
        "sample frequency:\n",
        "\n",
        "* Class weights\n",
        "* Sample weights"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f234a9a75b6d"
      },
      "source": [
        "### Class weights\n",
        "\n",
        "This is set by passing a dictionary to the `class_weight` argument to\n",
        "`Model.fit()`. This dictionary maps class indices to the weight that should\n",
        "be used for samples belonging to this class.\n",
        "\n",
        "This can be used to balance classes without resampling, or to train a\n",
        "model that gives more importance to a particular class.\n",
        "\n",
        "For instance, if class \"0\" is half as represented as class \"1\" in your data,\n",
        "you could use `Model.fit(..., class_weight={0: 1., 1: 0.5})`."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9929d26d91b8"
      },
      "source": [
        "Here's a NumPy example where we use class weights or sample weights to\n",
        "give more importance to the correct classification of class #5 (which\n",
        "is the digit \"5\" in the MNIST dataset)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f1844f2329a6"
      },
      "outputs": [],
      "source": [
        "import numpy as np\n",
        "\n",
        "class_weight = {\n",
        "    0: 1.0,\n",
        "    1: 1.0,\n",
        "    2: 1.0,\n",
        "    3: 1.0,\n",
        "    4: 1.0,\n",
        "    # Set weight \"2\" for class \"5\",\n",
        "    # making this class 2x more important\n",
        "    5: 2.0,\n",
        "    6: 1.0,\n",
        "    7: 1.0,\n",
        "    8: 1.0,\n",
        "    9: 1.0,\n",
        "}\n",
        "\n",
        "print(\"Fit with class weight\")\n",
        "model = get_compiled_model()\n",
        "model.fit(x_train, y_train, class_weight=class_weight, batch_size=64, epochs=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ce27221fad08"
      },
      "source": [
        "### Sample weights\n",
        "\n",
        "For fine grained control, or if you are not building a classifier,\n",
        "you can use \"sample weights\".\n",
        "\n",
        "- When training from NumPy data: Pass the `sample_weight`\n",
        "  argument to `Model.fit()`.\n",
        "- When training from `tf.data` or any other sort of iterator:\n",
        "  Yield `(input_batch, label_batch, sample_weight_batch)` tuples.\n",
        "\n",
        "A \"sample weights\" array is an array of numbers that specify how much weight\n",
        "each sample in a batch should have in computing the total loss. It is commonly\n",
        "used in imbalanced classification problems (the idea being to give more weight\n",
        "to rarely-seen classes).\n",
        "\n",
        "When the weights used are ones and zeros, the array can be used as a *mask* for\n",
        "the loss function (entirely discarding the contribution of certain samples to\n",
        "the total loss)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f9819d647793"
      },
      "outputs": [],
      "source": [
        "sample_weight = np.ones(shape=(len(y_train),))\n",
        "sample_weight[y_train == 5] = 2.0\n",
        "\n",
        "print(\"Fit with sample weight\")\n",
        "model = get_compiled_model()\n",
        "model.fit(x_train, y_train, sample_weight=sample_weight, batch_size=64, epochs=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eae5837c5f56"
      },
      "source": [
        "Here's a matching `Dataset` example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "c870f3f0c66c"
      },
      "outputs": [],
      "source": [
        "sample_weight = np.ones(shape=(len(y_train),))\n",
        "sample_weight[y_train == 5] = 2.0\n",
        "\n",
        "# Create a Dataset that includes sample weights\n",
        "# (3rd element in the return tuple).\n",
        "train_dataset = tf.data.Dataset.from_tensor_slices((x_train, y_train, sample_weight))\n",
        "\n",
        "# Shuffle and slice the dataset.\n",
        "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
        "\n",
        "model = get_compiled_model()\n",
        "model.fit(train_dataset, epochs=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3963bfa348b0"
      },
      "source": [
        "## Passing data to multi-input, multi-output models\n",
        "\n",
        "In the previous examples, we were considering a model with a single input (a tensor of\n",
        "shape `(764,)`) and a single output (a prediction tensor of shape `(10,)`). But what\n",
        "about models that have multiple inputs or outputs?\n",
        "\n",
        "Consider the following model, which has an image input of shape `(32, 32, 3)` (that's\n",
        "`(height, width, channels)`) and a timeseries input of shape `(None, 10)` (that's\n",
        "`(timesteps, features)`). Our model will have two outputs computed from the\n",
        "combination of these inputs: a \"score\" (of shape `(1,)`) and a probability\n",
        "distribution over five classes (of shape `(5,)`)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5f958449a057"
      },
      "outputs": [],
      "source": [
        "image_input = keras.Input(shape=(32, 32, 3), name=\"img_input\")\n",
        "timeseries_input = keras.Input(shape=(None, 10), name=\"ts_input\")\n",
        "\n",
        "x1 = layers.Conv2D(3, 3)(image_input)\n",
        "x1 = layers.GlobalMaxPooling2D()(x1)\n",
        "\n",
        "x2 = layers.Conv1D(3, 3)(timeseries_input)\n",
        "x2 = layers.GlobalMaxPooling1D()(x2)\n",
        "\n",
        "x = layers.concatenate([x1, x2])\n",
        "\n",
        "score_output = layers.Dense(1, name=\"score_output\")(x)\n",
        "class_output = layers.Dense(5, name=\"class_output\")(x)\n",
        "\n",
        "model = keras.Model(\n",
        "    inputs=[image_input, timeseries_input], outputs=[score_output, class_output]\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "df3ed34fe78b"
      },
      "source": [
        "Let's plot this model, so you can clearly see what we're doing here (note that the\n",
        "shapes shown in the plot are batch shapes, rather than per-sample shapes)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ac8c1baca9e3"
      },
      "outputs": [],
      "source": [
        "keras.utils.plot_model(model, \"multi_input_and_output_model.png\", show_shapes=True)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4d979e89b335"
      },
      "source": [
        "At compilation time, we can specify different losses to different outputs, by passing\n",
        "the loss functions as a list:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "9655c0084d70"
      },
      "outputs": [],
      "source": [
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
        "    loss=[keras.losses.MeanSquaredError(), keras.losses.CategoricalCrossentropy()],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "f5fc73405283"
      },
      "source": [
        "If we only passed a single loss function to the model, the same loss function would be\n",
        "applied to every output (which is not appropriate here).\n",
        "\n",
        "Likewise for metrics:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b4c0c6c564bc"
      },
      "outputs": [],
      "source": [
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
        "    loss=[keras.losses.MeanSquaredError(), keras.losses.CategoricalCrossentropy()],\n",
        "    metrics=[\n",
        "        [\n",
        "            keras.metrics.MeanAbsolutePercentageError(),\n",
        "            keras.metrics.MeanAbsoluteError(),\n",
        "        ],\n",
        "        [keras.metrics.CategoricalAccuracy()],\n",
        "    ],\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4dd9fb0343cc"
      },
      "source": [
        "Since we gave names to our output layers, we could also specify per-output losses and\n",
        "metrics via a dict:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "42cb75110fc3"
      },
      "outputs": [],
      "source": [
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
        "    loss={\n",
        "        \"score_output\": keras.losses.MeanSquaredError(),\n",
        "        \"class_output\": keras.losses.CategoricalCrossentropy(),\n",
        "    },\n",
        "    metrics={\n",
        "        \"score_output\": [\n",
        "            keras.metrics.MeanAbsolutePercentageError(),\n",
        "            keras.metrics.MeanAbsoluteError(),\n",
        "        ],\n",
        "        \"class_output\": [keras.metrics.CategoricalAccuracy()],\n",
        "    },\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bfd95ac0dd8b"
      },
      "source": [
        "We recommend the use of explicit names and dicts if you have more than 2 outputs.\n",
        "\n",
        "It's possible to give different weights to different output-specific losses (for\n",
        "instance, one might wish to privilege the \"score\" loss in our example, by giving to 2x\n",
        "the importance of the class loss), using the `loss_weights` argument:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "23a71e5f5227"
      },
      "outputs": [],
      "source": [
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
        "    loss={\n",
        "        \"score_output\": keras.losses.MeanSquaredError(),\n",
        "        \"class_output\": keras.losses.CategoricalCrossentropy(),\n",
        "    },\n",
        "    metrics={\n",
        "        \"score_output\": [\n",
        "            keras.metrics.MeanAbsolutePercentageError(),\n",
        "            keras.metrics.MeanAbsoluteError(),\n",
        "        ],\n",
        "        \"class_output\": [keras.metrics.CategoricalAccuracy()],\n",
        "    },\n",
        "    loss_weights={\"score_output\": 2.0, \"class_output\": 1.0},\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "367f598029e7"
      },
      "source": [
        "You could also chose not to compute a loss for certain outputs, if these outputs meant\n",
        "for prediction but not for training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6d51aa372ef4"
      },
      "outputs": [],
      "source": [
        "# List loss version\n",
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
        "    loss=[None, keras.losses.CategoricalCrossentropy()],\n",
        ")\n",
        "\n",
        "# Or dict loss version\n",
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
        "    loss={\"class_output\": keras.losses.CategoricalCrossentropy()},\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8314a8b3a7c7"
      },
      "source": [
        "Passing data to a multi-input or multi-output model in fit works in a similar way as\n",
        "specifying a loss function in compile: you can pass **lists of NumPy arrays** (with\n",
        "1:1 mapping to the outputs that received a loss function) or **dicts mapping output\n",
        "names to NumPy arrays**."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "0539da84328b"
      },
      "outputs": [],
      "source": [
        "model.compile(\n",
        "    optimizer=keras.optimizers.RMSprop(1e-3),\n",
        "    loss=[keras.losses.MeanSquaredError(), keras.losses.CategoricalCrossentropy()],\n",
        ")\n",
        "\n",
        "# Generate dummy NumPy data\n",
        "img_data = np.random.random_sample(size=(100, 32, 32, 3))\n",
        "ts_data = np.random.random_sample(size=(100, 20, 10))\n",
        "score_targets = np.random.random_sample(size=(100, 1))\n",
        "class_targets = np.random.random_sample(size=(100, 5))\n",
        "\n",
        "# Fit on lists\n",
        "model.fit([img_data, ts_data], [score_targets, class_targets], batch_size=32, epochs=1)\n",
        "\n",
        "# Alternatively, fit on dicts\n",
        "model.fit(\n",
        "    {\"img_input\": img_data, \"ts_input\": ts_data},\n",
        "    {\"score_output\": score_targets, \"class_output\": class_targets},\n",
        "    batch_size=32,\n",
        "    epochs=1,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "e53eda8e1399"
      },
      "source": [
        "Here's the `Dataset` use case: similarly as what we did for NumPy arrays, the `Dataset`\n",
        "should return a tuple of dicts."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4df41a12ed2c"
      },
      "outputs": [],
      "source": [
        "train_dataset = tf.data.Dataset.from_tensor_slices(\n",
        "    (\n",
        "        {\"img_input\": img_data, \"ts_input\": ts_data},\n",
        "        {\"score_output\": score_targets, \"class_output\": class_targets},\n",
        "    )\n",
        ")\n",
        "train_dataset = train_dataset.shuffle(buffer_size=1024).batch(64)\n",
        "\n",
        "model.fit(train_dataset, epochs=1)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "38ebf30ce6ac"
      },
      "source": [
        "## Using callbacks\n",
        "\n",
        "Callbacks in Keras are objects that are called at different points during training (at\n",
        "the start of an epoch, at the end of a batch, at the end of an epoch, etc.) and which\n",
        "can be used to implement behaviors such as:\n",
        "\n",
        "- Doing validation at different points during training (beyond the built-in per-epoch\n",
        "validation)\n",
        "- Checkpointing the model at regular intervals or when it exceeds a certain accuracy\n",
        "threshold\n",
        "- Changing the learning rate of the model when training seems to be plateauing\n",
        "- Doing fine-tuning of the top layers when training seems to be plateauing\n",
        "- Sending email or instant message notifications when training ends or where a certain\n",
        "performance threshold is exceeded\n",
        "- Etc.\n",
        "\n",
        "Callbacks can be passed as a list to your call to `fit()`:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "15036ddbee42"
      },
      "outputs": [],
      "source": [
        "model = get_compiled_model()\n",
        "\n",
        "callbacks = [\n",
        "    keras.callbacks.EarlyStopping(\n",
        "        # Stop training when `val_loss` is no longer improving\n",
        "        monitor=\"val_loss\",\n",
        "        # \"no longer improving\" being defined as \"no better than 1e-2 less\"\n",
        "        min_delta=1e-2,\n",
        "        # \"no longer improving\" being further defined as \"for at least 2 epochs\"\n",
        "        patience=2,\n",
        "        verbose=1,\n",
        "    )\n",
        "]\n",
        "model.fit(\n",
        "    x_train,\n",
        "    y_train,\n",
        "    epochs=20,\n",
        "    batch_size=64,\n",
        "    callbacks=callbacks,\n",
        "    validation_split=0.2,\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "303815509732"
      },
      "source": [
        "### Many built-in callbacks are available\n",
        "\n",
        "- `ModelCheckpoint`: Periodically save the model.\n",
        "- `EarlyStopping`: Stop training when training is no longer improving the validation\n",
        "metrics.\n",
        "- `TensorBoard`: periodically write model logs that can be visualized in\n",
        "[TensorBoard](https://www.tensorflow.org/tensorboard) (more details in the section\n",
        "\"Visualization\").\n",
        "- `CSVLogger`: streams loss and metrics data to a CSV file.\n",
        "- etc.\n",
        "\n",
        "See the [callbacks documentation](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/) for the complete list.\n",
        "\n",
        "### Writing your own callback\n",
        "\n",
        "You can create a custom callback by extending the base class\n",
        "`keras.callbacks.Callback`. A callback has access to its associated model through the\n",
        "class property `self.model`.\n",
        "\n",
        "Make sure to read the\n",
        "[complete guide to writing custom callbacks](https://www.tensorflow.org/guide/keras/custom_callback/).\n",
        "\n",
        "Here's a simple example saving a list of per-batch loss values during training:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "b265d36ce608"
      },
      "outputs": [],
      "source": [
        "class LossHistory(keras.callbacks.Callback):\n",
        "    def on_train_begin(self, logs):\n",
        "        self.per_batch_losses = []\n",
        "\n",
        "    def on_batch_end(self, batch, logs):\n",
        "        self.per_batch_losses.append(logs.get(\"loss\"))\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "5ee672524987"
      },
      "source": [
        "## Checkpointing models\n",
        "\n",
        "When you're training model on relatively large datasets, it's crucial to save\n",
        "checkpoints of your model at frequent intervals.\n",
        "\n",
        "The easiest way to achieve this is with the `ModelCheckpoint` callback:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "83614be57725"
      },
      "outputs": [],
      "source": [
        "model = get_compiled_model()\n",
        "\n",
        "callbacks = [\n",
        "    keras.callbacks.ModelCheckpoint(\n",
        "        # Path where to save the model\n",
        "        # The two parameters below mean that we will overwrite\n",
        "        # the current checkpoint if and only if\n",
        "        # the `val_loss` score has improved.\n",
        "        # The saved model name will include the current epoch.\n",
        "        filepath=\"mymodel_{epoch}\",\n",
        "        save_best_only=True,  # Only save a model if `val_loss` has improved.\n",
        "        monitor=\"val_loss\",\n",
        "        verbose=1,\n",
        "    )\n",
        "]\n",
        "model.fit(\n",
        "    x_train, y_train, epochs=2, batch_size=64, callbacks=callbacks, validation_split=0.2\n",
        ")"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7f6afa36950c"
      },
      "source": [
        "The `ModelCheckpoint` callback can be used to implement fault-tolerance:\n",
        "the ability to restart training from the last saved state of the model in case training\n",
        "gets randomly interrupted. Here's a basic example:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "27ce92b2ad58"
      },
      "outputs": [],
      "source": [
        "import os\n",
        "\n",
        "# Prepare a directory to store all the checkpoints.\n",
        "checkpoint_dir = \"./ckpt\"\n",
        "if not os.path.exists(checkpoint_dir):\n",
        "    os.makedirs(checkpoint_dir)\n",
        "\n",
        "\n",
        "def make_or_restore_model():\n",
        "    # Either restore the latest model, or create a fresh one\n",
        "    # if there is no checkpoint available.\n",
        "    checkpoints = [checkpoint_dir + \"/\" + name for name in os.listdir(checkpoint_dir)]\n",
        "    if checkpoints:\n",
        "        latest_checkpoint = max(checkpoints, key=os.path.getctime)\n",
        "        print(\"Restoring from\", latest_checkpoint)\n",
        "        return keras.models.load_model(latest_checkpoint)\n",
        "    print(\"Creating a new model\")\n",
        "    return get_compiled_model()\n",
        "\n",
        "\n",
        "model = make_or_restore_model()\n",
        "callbacks = [\n",
        "    # This callback saves a SavedModel every 100 batches.\n",
        "    # We include the training loss in the saved model name.\n",
        "    keras.callbacks.ModelCheckpoint(\n",
        "        filepath=checkpoint_dir + \"/ckpt-loss={loss:.2f}\", save_freq=100\n",
        "    )\n",
        "]\n",
        "model.fit(x_train, y_train, epochs=1, callbacks=callbacks)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "da3ab58d5235"
      },
      "source": [
        "You call also write your own callback for saving and restoring models.\n",
        "\n",
        "For a complete guide on serialization and saving, see the\n",
        "[guide to saving and serializing Models](https://www.tensorflow.org/guide/keras/save_and_serialize/)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b9342cc2ddba"
      },
      "source": [
        "## Using learning rate schedules\n",
        "\n",
        "A common pattern when training deep learning models is to gradually reduce the learning\n",
        "as training progresses. This is generally known as \"learning rate decay\".\n",
        "\n",
        "The learning decay schedule could be static (fixed in advance, as a function of the\n",
        "current epoch or the current batch index), or dynamic (responding to the current\n",
        "behavior of the model, in particular the validation loss).\n",
        "\n",
        "### Passing a schedule to an optimizer\n",
        "\n",
        "You can easily use a static learning rate decay schedule by passing a schedule object\n",
        "as the `learning_rate` argument in your optimizer:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "684f0ab6d3de"
      },
      "outputs": [],
      "source": [
        "initial_learning_rate = 0.1\n",
        "lr_schedule = keras.optimizers.schedules.ExponentialDecay(\n",
        "    initial_learning_rate, decay_steps=100000, decay_rate=0.96, staircase=True\n",
        ")\n",
        "\n",
        "optimizer = keras.optimizers.RMSprop(learning_rate=lr_schedule)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "7d742e44f535"
      },
      "source": [
        "Several built-in schedules are available: `ExponentialDecay`, `PiecewiseConstantDecay`,\n",
        "`PolynomialDecay`, and `InverseTimeDecay`.\n",
        "\n",
        "### Using callbacks to implement a dynamic learning rate schedule\n",
        "\n",
        "A dynamic learning rate schedule (for instance, decreasing the learning rate when the\n",
        "validation loss is no longer improving) cannot be achieved with these schedule objects\n",
        "since the optimizer does not have access to validation metrics.\n",
        "\n",
        "However, callbacks do have access to all metrics, including validation metrics! You can\n",
        "thus achieve this pattern by using a callback that modifies the current learning rate\n",
        "on the optimizer. In fact, this is even built-in as the `ReduceLROnPlateau` callback."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b4a05f880175"
      },
      "source": [
        "## Visualizing loss and metrics during training\n",
        "\n",
        "The best way to keep an eye on your model during training is to use\n",
        "[TensorBoard](https://www.tensorflow.org/tensorboard), a browser-based application\n",
        "that you can run locally that provides you with:\n",
        "\n",
        "- Live plots of the loss and metrics for training and evaluation\n",
        "- (optionally) Visualizations of the histograms of your layer activations\n",
        "- (optionally) 3D visualizations of the embedding spaces learned by your `Embedding`\n",
        "layers\n",
        "\n",
        "If you have installed TensorFlow with pip, you should be able to launch TensorBoard\n",
        "from the command line:\n",
        "\n",
        "```\n",
        "tensorboard --logdir=/full_path_to_your_logs\n",
        "```"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1fcf386a1dad"
      },
      "source": [
        "### Using the TensorBoard callback\n",
        "\n",
        "The easiest way to use TensorBoard with a Keras model and the fit method is the\n",
        "`TensorBoard` callback.\n",
        "\n",
        "In the simplest case, just specify where you want the callback to write logs, and\n",
        "you're good to go:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "f74247282ff6"
      },
      "outputs": [],
      "source": [
        "keras.callbacks.TensorBoard(\n",
        "    log_dir=\"/full_path_to_your_logs\",\n",
        "    histogram_freq=0,  # How often to log histogram visualizations\n",
        "    embeddings_freq=0,  # How often to log embedding visualizations\n",
        "    update_freq=\"epoch\",\n",
        ")  # How often to write logs (default: once per epoch)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "50cd5f8631fd"
      },
      "source": [
        "For more information, see the\n",
        "[documentation for the `TensorBoard` callback](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/tensorboard/)."
      ]
    }
  ],
  "metadata": {
    "colab": {
      "collapsed_sections": [],
      "name": "train_and_evaluate.ipynb",
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
